The present invention relates to image data processing, and more particularly, to an image data post-processing method for reducing quantization effect, and an apparatus therefor.
The present invention is adopted as ISO/IEC JTC1/SC29/WG11 N1902 (ISO/IEC 14496-2 Committee Draft).
Generally, picture encoding standards such as MPEG of the International Standardization Organization (ISO) and H.263 recommended by the International Telecommunication Union (ITU) adopt block-based motion estimation and discrete cosine transform (DCT) of blocks. Also, most video coding standards use an 8xc3x978 pixel block DCT for packing information into a few transform coefficients. Such block-based DCT schemes take advantage of the local spatial correlation property of images.
However, when image data which has been coded based on blocks is restored, the restored image is considerably deteriorated, causing blocking artifacts near block boundaries, corner outliers at cross points of blocks, and ringing noise near the image edge. Because MPEG quantizes the transformed coefficient of 8xc3x978 pixel block. In particular, when the image is highly compressed, the deterioration of the image becomes serious.
When an image is highly compressed, the block-based coding induces the well-known blocking artifacts near the block boundary, corner outliers at the cross points of the blocks, and ringing noise near the image edge.
The blocking artifacts are grid noise occurring along the block boundaries in a relatively homogeneous area. The grid noise shows traces of the block-based process at the edges between blocks when the compressed data is displayed on a screen after being restored. Thus, the edges between blocks are identified. Also, corner outliers occur at the corner points of the 8xc3x978 block. Also, the ringing noise is a typical Gibb""s phenomenon occurring by truncation when high-frequency coefficients of the DCT are quantized so as to highly compress the image. As a result, overlapping of image with a predetermined interval due to the ringing noise is noticeable.
Several methods for reducing the blocking artifacts, corner outliers and ringing noise, caused by block-based coding, have been suggested in the following articles: [1] Lee, H. C. Kim and H. W. Park, xe2x80x9cBlocking Effect Reduction of JPEG images by Signal Adaptive Filteringxe2x80x9d, in press IEEE Trans. on Image processing, 1997, [2] B. Ramanurthi and A. Gersho, xe2x80x9cNonlinear Space Variant Postprocessing of Block Coded Imagesxe2x80x9d, IEEE Trans. on ASSP, Vol. 34, No. 5, pp. 1258-1267, 1986, [3] Y. Ynag, N. Galatsanos and A. Katsaggelos, xe2x80x9cProjection-Based Spatially Adaptive Reconstruction of Block-Transform Compressed Imagesxe2x80x9d, IEEE Trans. on Image processing, Vol. 4, No. 7, pp. 896-908, Jul. 1995, [4] Z. Xiong, M. T. Orchard, and Y. Q. Zhang, xe2x80x9cA Deblocking Algorithum for JPEG Compressed Images Using Overcomplete Wavelet Representationsxe2x80x9d, IEEE Trans. Circuits Syst. Video Technol., Vol. 7, No. 2, pp. 433-437, 1997.
In the reference [1], 2-dimensional signal-adaptive filtering (SAF) for reducing quantization effect of a JPEG-decompressed image has been suggested. Also, in the reference [2], a 2-dimensional filter is used to reduce the blocking artifacts and a 1-imensional filter is used to reduce staircase noise, resulting in good effects. In the reference [3], an iterative image-recovery algorithm using the theory of projections onto convex sets (POCS) has also been proposed.
However, the main drawback of these algorithms is their computation complexity. Meanwhile, post-filtering which uses over-complete wavelet representation which reduces the complexity of computation has been suggested by the reference [4].
However, the post-filtering method is applied only to JPEG-decompressed images. Also, for low bit-rate coding, a spatio-temporal adaptive post-filtering which can be applied to a 3-dimensional subband coding has been suggested in the reference [5]: T. S. Liu and N. Jayant, xe2x80x9cAdaptive Postprocessing Algorithms for Low Bit Rate Video Signalsxe2x80x9d, IEEE Trans. on Image Processing, Vol. 4, No. 7, pp. 1032-1035, Jul. 1995. However, this method also has computation complexity.
To solve the above problems, it is an objective of the present invention to provide an image data post-processing method for reducing quantization effects such as blocking artifacts, corner outliers and ringing noise, from an MPEG-decompressed image, which can perform low bit rate coding without complex computation, and an apparatus therefor.
According to an aspect of the present invention, there is provided an image data post-processing method for reducing quantization effect induced when image data compressed based on a block is decoded, the method comprising the steps of: (a) detecting semaphore representing whether or not post-processing is required, using distribution of inverse quantization coefficients of inverse-quantized image data and a motion vector representing the difference between the blocks of a previous video object plane (VOP) and blocks of a current VOP; and (b) filtering the decoded image data corresponding to the semaphore by a predetermined method, if it is determined by checking the detected semaphore that post-processing is required.
Preferably, the step (a) is performed on an intra-VOP in an intra-VOP mode, and on in inter-VOP in an inter-VOP mode.
Preferably, the semaphore includes a blocking semaphore representing whether or not reduction of blocking artifacts near block boundaries is required, and a ringing semaphore representing whether or not reduction of ringing noise near image edges is required.
Preferably, the blocking semaphore and the ringing semaphore of the intra-VOP are detected by investigating distribution of inverse quantization coefficients which are discrete cosine transform (DCT) coefficients after the compressed bitstream is inverse quantized.
Preferably, the blocking semaphore of the intra-VOP comprises a horizontal blocking semaphore (HBS) and a vertical blocking semaphore (VBS), assuming that the uppermost and leftmost pixel of the block, among 64 pixels constituting the 8xc3x978 block, is pixel A, the pixel to the right of the pixel A is a pixel B, and the pixel below the pixel A is a pixel C, and the HBS and the VBS of the intra-VOP are extracted by the steps of: (a) calculating discrete cosine transform (DCT) coefficients on the inverse-quantized 8xc3x978 block after the compressed image data is inversely quantized; (b) setting the HBS and the VBS to xe2x80x9c1xe2x80x9d which means that post-processing is required, if only the coefficient of the pixel A is non-zero; (c) setting the VBS to xe2x80x9c1xe2x80x9d which means that post-processing is required, if only the top row of the inverse-quantized 8xc3x978 block includes non-zero coefficient pixel; and (d) setting the HBS to xe2x80x9c1xe2x80x9d which means that the post-process is required, if on the leftmost column of the inverse-quantized 8xc3x978 block includes non-zero coefficient pixel.
Preferably, assuming that the uppermost and leftmost pixel of the block, among 64 pixels constituting the 8xc3x978 block, is pixel A, the pixel to the right of the pixel A is a pixel B, and the pixel below the pixel A is a pixel C, the ringing semaphore (RS) of the intra-VOP is set to xe2x80x9c1xe2x80x9d which means post-processing is required, if any pixel other than the pixels A, B and C of the inverse-quantized 8xc3x978 block has a non-zero coefficient.
Preferably, the blocking semaphore of the current inter-VOP comprises a horizontal blocking semaphore (HBS) and a vertical blocking semaphore (VBS), and assuming that a reference VOP comprises predetermined reference blocks, and the block of the reference VOP predicted by a motion vector (MVx,MVy) of a block Ac of the current inter-VOP is a motion block X, the HBS and the VBS of on the block Ac of the current inter-VOP are extracted by the steps of: checking the degree of overlapping between the motion block X and the reference blocks; performing a bit-wise AND operation on the HBS and VBS of the reference blocks in which the number of the overlapped pixels is more than a predetermined number; and setting the HBS and the VBS of the block Ac of the current VOP to the result of the operation.
Preferably, assuming that a reference VOP comprises predetermined reference blocks, and the block of the reference VOP predicted by a motion vector (MVx,MVy) of a block Ac of the current inter-VOP is a motion block X, the ringing semaphore (RS) of the block Ac of the current inter-VOP is extracted by the steps of: setting the RS of the current block Ac to xe2x80x9c1xe2x80x9d if an inverse quantized coefficient (IQC) of a residual signal in the 8xc3x978 block of the inter-VOP is non-zero; setting the RS of the block to xe2x80x9c1xe2x80x9d in an 8xc3x978 prediction mode which is supported by the MPEG4 algorithm and transfers four motion vectors on one macroblock (MB); and checking the degree of overlapping between the motion block X and the reference blocks, if the RS is still zero, and performing a bit-wise OR operation on the RS of the reference blocks in which the number of the overlapped pixels is more than a predetermined number, to set the RS of the block Ac of the current VOP to the result of the operation.
Preferably, filtering is performed by the steps of: (a) changing a predetermined number of pixel values of a horizontal block boundary between a block I and a block J adjacent to the block I if the HBSs of the blocks I and J are set to xe2x80x9c1xe2x80x9d; (b) comparing the difference between the values of two pixels adjacent to each other around the horizontal block boundary with a quantization factor (QP) of the H.263 if the HBS of either the block I or the block J is zero, and changing the values of the pixels whose number is less than in the step (a) if the difference of the pixels is less than the QP, wherein the filtering on the pixels around the vertical block boundary is performed using the VBS in the same manner as in the pixels around the horizontal block boundary.
Preferably, assuming six pixels around the horizontal block boundary between the blocks I and J are pixels A, B, C, D, E and F, the pixels C and D are nearest to the horizontal block boundary, the pixels A and F are farthest to the horizontal block boundary, and the pixels B and E are located at the middle of the pixels A and C, and pixels D and F, low-pass filtering on the 6 pixels is performed using a 7-tab (1,1,1,2,1,1,1) low pass filter in the step (a), and the filtering of the step (b) is performed on the pixels B, C, D and E, wherein assuming that the difference between the pixels C and D is d, the pixels C and D are filtered as an average of the pixels C and D, and the filtered pixels B and E are different from the pixels B and E, respectively, by d/8.
Preferably, the filtering step comprises the steps of: detecting horizontal and vertical edges of image data; and performing 2-dimensional adaptive signal filtering on an 8xc3x978 block requiring reduction of ringing noise, wherein assuming that a pixel within a block having a predetermined size is pixel[m][n], the pixel to the right of the pixel[m][n] is pixel[m][n+1], the pixel to the left of the pixel[m][n] is pixel[m][nxe2x88x921], the difference between the pixel[m][n] and the pixel[m][n+1] is A1, and the difference between the pixel[m][n] and the pixel[m][n+1] is A2, and the quantization factor of the H.263 is QP, the horizontal edge detection is performed by a logical formula ((A1 greater than QP) and (A2 greater than QP)) or (A1 greater than 2QP) or (A2 greater than 2QP) wherein the pixel[m][n] is determined as edge and edge map Edge[m][n] is set to xe2x80x9c1xe2x80x9d if the logical formula is satisfied, and assuming that the pixel above the pixel[m][n] is pixel[mxe2x88x921][n], the pixel below the pixel[m][n] is pixel[mxe2x88x921][n], the difference between the pixel[m][n] and the pixel[m+1][n] is Axe2x80x21, the difference between the pixel[m][n] and the pixel[mxe2x88x921][n] is Axe2x80x22, and the quantization factor of the H.263 is QP, the vertical edge detection is performed by a logical formula ((Axe2x80x21 greater than QP) and (Axe2x80x22 greater than QP)) or (Axe2x80x21 greater than 2QP) or (Axe2x80x22 greater than 2QP) wherein the pixel[m][n] is determined as edge and edge map Edge[m][n] is set to xe2x80x9c1xe2x80x9d if the logical formula is satisfied, and signal adaptive filtering is performed by applying a 4-connected filter window to the 8xc3x978 block, wherein filtering is not performed if the central pixel of the filter window is an edge, and weighted filtering is performed if the central pixel of the filter window is a non-edge pixel.
Also, there is provided an image data post-processing method for reducing corner outliers occurring at the corner of a cross point where four blocks meet when image data compressed based on a block are decoded, the method comprising the steps of: (a) detecting corner outliers from the block of inverse-quantized image data; and (b) compensating for the detected corner outliers. Assuming that four pixels around the cross point are pixels A, B, C and D, value[0] is A, value[1] is B, value[2] is C, value[3] is D, (A+B+C+D+2)/4 is Average, A1 and A2 are pixels adjacent to the pixel A in the block to which the pixel A belongs, A3 is a pixel diagonal to the pixel A, B1 and B2 are pixels adjacent to the pixel B in the block to which the pixel B belongs, B3 is a pixel diagonal to the pixel B, C, and C2 are pixels adjacent to the pixel C in the block to which the pixel C belongs, C3 is a pixel diagonal to the pixel C, D1 and D2 are pixels adjacent to the pixel D in the block to which the pixel D belongs, and D3 is a pixel diagonal to the pixel D, the step (a) may comprise the sub-steps of: (a1) comparing the difference between the value[0] and the Average with the quantization factor (QP) of the H.263, and counting the pixel A as the corner outlier candidate pixel if the difference is greater than the QP; (a2) performing the step (a1) on the value[1], value[2] and value[3] to count the corresponding pixel as a corner outlier candidate pixel; and (a3) detecting the pixel as a corner outlier pixel if the corner outlier candidate pixel is only one, and detecting the candidate pixel having the greatest difference from (A3+B3+C3+D3+2)/4 as a corner outlier pixel if there are two or more corner outlier candidate pixels. Also, the step (b) may be performed by compensating for the pixel A as Axe2x80x2 by (4A+B+C+2D+4)/8, the pixel A1 as A1xe2x80x2 by (Axe2x80x2+3A1+2)/4, the pixel A2 as A2xe2x80x2 by (Axe2x80x2+3A2+2)/4, if the corner outlier pixel is A and the difference between the pixels A and A3 is less than 3QP/2; compensating for the pixel B as Bxe2x80x2 by (4B+C+D+2A+4)/8, the pixel B1 as B1xe2x80x2 by (Bxe2x80x2+3B+2)/4, the pixel B2 as B2xe2x80x2 by (Bxe2x80x2+BA2+2)/4, if the corner outlier pixel is B and the difference between the pixels B and B3 is less than 3QP/2; compensating for the pixel C as Cxe2x80x2 by (4C+D+A+2B+4)/8, the pixel C1 as C1xe2x80x2 by (Cxe2x80x2+3C1+2)/4, the pixel C2 a C2xe2x80x2 by (Cxe2x80x2+3C2+2)/4, if the corner outlier pixel is C and the difference between the pixels C and C3 is less than 3QP/2; and compensating for the pixel D as Dxe2x80x2 by (4D+A+B+2C+4)/8, the pixel D1 as D1xe2x80x2 by (Dxe2x80x2+3D1+2)/4, the pixel D1as D2xe2x80x2 by (Dxe2x80x2+3D2+2)/4, if the corner outlier pixel is D and the difference between the pixels D and D3 is less than 3QP/2.
Also, there is provided an image data post-processing method for reducing quantization effect induced when image data compressed based on a block is decoded, the method comprising the steps of: (a) detecting semaphore representing whether or not post-processing is required, using distribution of inverse quantization coefficients of inverse-quantized image data and a motion vector representing the difference between blocks of previous video object plane (VOP) and blocks of a current VOP; (b) detecting a corner outlier pixel of the inverse-quantized image data block, by the above steps used in the above image data post-processing method for reducing the corner outliers; (c) filtering the decoded image data corresponding to the semaphore by a predetermined method, if it is determined by checking the detected semaphore that the post-process is required; and (d) compensating for the detected corner outlier through the steps used in the above image data post-processing method for reducing the corner outliers.
According to another aspect of the present invention, there is provided an image data post-processing apparatus for reducing quantization effect induced when image data compressed based on a block is decoded, the apparatus comprising: a semaphore detector for detecting a semaphore representing whether or not post-processing is required, using distribution of inverse quantization coefficients of inverse-quantized image data and a motion vector representing the difference between blocks of previous video object plane (VOP) and blocks of a current VOP; a deblocking filter for checking blocking semaphore detected by the semaphore detector and performing deblocking filtering on the decoded image data; a corner outlier compensator for detecting a corner outlier from the data passed through the deblocking filtering and compensating for the detected corner outlier; and a deringing filter for checking ringing semaphore detected by the semaphore detector and performing deringing filtering on the corner outlier compensated data.
Also, the present invention may be embodied as a program capable of being run by a computer, and may be embodied in a general purpose digital computer that is running the program from a computer usable medium including but not limited to storage media such as magnetic storage media (e.g., ROM""s, floppy disks, hard disks, etc.), optically readable media (e.g., CD-ROMs, DVDs, etc.) and carrier waves (e.g., transmissions over the Internet).
According to still another aspect of the present invention, there is provided a computer readable medium having embodied thereon a computer program for image data post-processing for reducing quantization effect induced when image data compressed based on a block is decoded, wherein the image data post-processing comprises the steps of: (a) detecting semaphore representing whether or not post-processing is required, using distribution of inverse quantization coefficients of inverse-quantized image data and a motion vector representing the difference between the blocks of a previous video object plane (VOP) and blocks of a current VOP; and (b) filtering the decoded image data corresponding to the semaphore by a predetermined method, if it is determined by checking the detected semaphore that post-processing is required.